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ACQUISITION OF 3-D SCENES WITH A SINGLE HAND HELD CAMERA 

REFERENCE TO RELATED APPLICATIONS 

This application claims an invention which was disclosed in Provisional 
Application Number 60/138,965 , filed June 11, 1999 , entitled "ACQUISITION OF 3-D 
SCENES WITH A SINGLE HAND HELD CAMERA". The benefit under 35 
USC §1 19(e) of the United States provisional application is hereby claimed, and the 
aforementioned application is hereby incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

FIELD OF THE INVENTION 

The invention pertains to the field of cameras. More particularly, the invention 
pertains to hand-held cameras and the acquisition of three-dimensional scenes. 

DESCRIPTION OF RELATED ART 

Stereographic photography is the method of producing images which are 
apparently three-dimensional by recording separate left- and right-eye images. The viewer 
reconstructs the 3-D image by viewing the two separate 2-D images simultaneously. 
Stereographic photography has been known since at least the late 19th century, when 
stereo viewers were a popular parlor accessory. 

Such stereo views have historically been created with two lenses on a single 
camera, spaced apart by approximately the inter-ocular distance of a human head. The 
Stereo Realist™ series of 35mm still cameras, popular in the 1950' s, are an example of this 
kind of imaging. Left and right views were recorded simultaneously through two 
lens/shutter sets on alternate frames of the 35mm film. The later Nimslo™ system used 
four lenses for essentially the same approach. 

Stereo movies appeared in the 1950' s. The images were typically created either 
using two synchronized cameras, or a two-lens system on a single camera. Similarly, the 
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various Stereo TV systems have typically used two cameras (see Iipton, et al, US Patent 
4,583,1 17) or a single camera with two lenses (Lipton, et al, US Patent 4,523,226). 

In the area of 3-D scene acquisition by stereo equipment, two signal processing 
tasks are involved: calibration of the camera pair [1] and estimation of corresponding 
pixels in the image pair [5]. 

In the area of camera calibration, there are two different techniques: fixed and self- 
calibration. In fixed calibration, all camera parameters are extracted off line by placing a 
special object with known geometry in front of the cameras and processing of the camera 
images [1, 2]. This method provides very accurate and complete results (all parameters are 
obtainable). Additionally, calibration reduces correspondence estimation from a 2-D 
search problem to a more efficient and reliable 1-D search [5]. 

However, fixed calibration suffers from a number of disadvantages. A special 
calibration object and user interaction is required. Each time the camera parameters 
change, e.g. due to zooming or change of convergence angle, the calibration has to be 
repeated. 

Self-calibration circumvents these disadvantages. In this method, the 
correspondences are estimated first, in an image pair of the scene. After this, the camera 
parameters are extracted from the found correspondence field [4]. The price to be paid is 
two-fold. First, correspondence estimation is a more demanding task since no reduction 
from a 2-D to a 1-D search can be applied. Secondly, in self -calibration methods with 
normal stereo camera pairs, there is no reference to the standard SI meter. Thus the scale 
of the 3-D models is not obtainable by the prior art methods [3]. 

SUMMARY OF THE INVENTION 

The acquisition of 3-D scenes by a single hand-held camera is disclosed. The 
camera is preferably mounted on a lens attachment with four mirrors, enabling stereo 
capturing of the scene. The signal processing tasks involved, camera calibration and 
correspondence estimation, are each disclosed. Both fixed and self -calibration methods 
benefit from the use of the lens attachment to acquire a scene. Specifically, full self- 
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calibration of the camera is enabled, without loss of absolute scale as seen in prior art 
stereo self-calibration methods. Accurate 3-D models are obtained with the method of the 
invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
5 Fig. 1 shows an example of an apparatus of the present invention. 

Fig. 2 shows the function of the mirrors in the apparatus of the present invention. 
Fig. 3 shows a stereo camera model. 

Fig. 4 shows a complete scheme of acquisition according to the present invention. 
Fig. 5a shows an image of the calibration plate of the present invention. 
10 Fig. 5b shows an image of an actual scene according to the present invention. 
Fig. 6 shows a rectified image pair according to the present invention. 
Fig. 7 shows a disparity field according to the present invention. 
Fig. 8 shows a 3-D model according to a method of the present invention. 

DETA ILED DESCRIPTION OF THE INVENTION 

15 The acquisition of 3-D scenes with a very specific stereo camera is disclosed. 

Using camera calibration parameters, two light rays originating from a pair of 
corresponding pixels are constructed. The intersection of the two rays then provides the 3- 
D coordinates of a scene point. 

The apparatus is shown in Fig. 1. A single hand held camera (1) is mounted on a 
20 lens attachment (2) with mirrors as disclosed in U.S. Patents 5,532,777, 5,828,913, and 
5,883,662, the disclosures of which are herein incorporated by reference [7-9]. Briefly, 
the lens attachment preferably has a set of four mirrors in two pairs located in front of a 
camera lens. The outer two mirrors (6) and (7) face generally outward along the optical 
axis of the lens and the inner two mirrors (3) and (4) face generally inward into the lens. 
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The centers of the outer two mirrors (6) and (7) are spaced apart by an appropriate 
interocular distance. The two inside mirrors (3) and (4) are together large enough to cover 
the complete viewing area of the lens, each taking exactly half of the lens viewing area. 
The two outside mirrors (6) and (7) are preferably bigger than the inside pair of mirrors (3) 
5 and (4) and large enough to cover the viewing area of the inside pair of mirrors (3) and (4) 
to avoid viewing area reduction. The convergence of the two outer mirrors (6) and (7) is 
adjustable by swiveling them simultaneously and equally about their centerlines with a 
ganging mechanism. The two center mirrors (3) and (4) may be fixed, or could be 
adjustable by being swiveled so that one side of each remains in tight contact with the 
10 other along the optical axis of the camera lens, and each makes a 45° or lesser angle to the 
optical axis. 

Imagine that the directions of the incoming light rays (5) are reversed, then the two 
center mirrors (3) and (4) split the bundle of light rays (5) from the camera in two parts. 
The two side mirrors (6) and (7) redirect each bundle towards the scene (8). The 
15 convergence point (9) of the two bundles is adjustable by rotation of the side mirrors (6) 
and (7). The image (10) is also shown. 

The small size and low weight of this stereo camera provide high user mobility. In 
addition, the use of a single camera is economical and does not require shutter 
synchronization of a camera pair. For storage, only one conventional recorder is needed. 
20 Both the calibration and correspondence estimation tasks benefit from this particular 
setup. 

Correspondence estimation is based on photometric similarity of corresponding 
pixels. Photometrically unbalanced stereo cameras are a cause of errors, which is avoided 
to a large extent by the use of a single camera. As discussed in the background, there are 
25 two different calibration techniques: fixed and self-calibration. Both of these types of 
calibrations benefit from the apparatus shown in Fig. 1. In prior art self-calibration 
methods with normal stereo camera pairs, there was no reference to the standard SI meter. 
Thus the scale of the 3-D models were not obtainable [3]. The known geometry of the 
apparatus of the present invention provides the scale. The use of a single camera in 
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concert with the lens attachment simplifies the stereo camera model without loss of 
generality. 

STEREO CAMERA MODEL 

The camera model for the device is a novel version of the general model for stereo 
5 cameras in [4]. Fig. 2 shows the function of the mirrors (13), (14), (16) and (17) in the lens 
attachment. The single real camera (11) is split into two virtual cameras (12) and (15), 
each with half of the original CCD chip. The half CCDs are not centered on the optical 
axes (20) and (21) of the virtual left (12) and right cameras (15). The rotation of the two 
side mirrors (16) and (17) is mechanically coupled. To have any overlap in the two virtual 
10 camera images, a = 45° + Acc , with Aa > 0 . If the side mirrors (16) and (17) are rotated 
around point P and Q, the two virtual cameras (12) and (15) rotate around the same points 
with double speed. 

Fig. 3 shows the general stereo camera model from [4]. Five reference frames are 
defined, the stereo frame (22), the left/right lens frames (23) and (24) and the left/right 
15 projection frames (25) and (26). The camera baseline is B. The frame SF (22) is defined to 
be a right handed frame in which the two optical centers lie on the x-axis symmetrically 
around the origin, at (-V4B, 0,0) for the left camera and (-P^B, 0,0) for the right camera, in 
SF coordinates. From Fig. 2, it is deduced that: 

B = 2w + 2 ( ft + w) sin 2 A a + £ 0 ( 1 ) 

20 relating meters to angles. This provides a means for self-calibration in meters, instead of 
unknown units. The 8o models remaining imperfections, and is assumed to be small. 

The orientations of the left and right lens frames are defined by two sets of Euler 
angles ((p x , (p y , (p z ). The lens is present in the origin of the lens frames LF-L and LF-R, 
oriented in their xy planes. Radial symmetry in the lenses is assumed. At this point, the 
25 angle cp z has no meaning. Instead of discarding <p z , it is used for the orientation of the 
CCD. The reference frame SF (22) is defined up to a rotation around the *-axis. An 
arbitrary equation is introduced that eliminates either <p x;L or <p x; R, such as (p x;L + <p*;R = °- 
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Ideally, both are zero, but due to imperfections in the apparatus and the hand-cam this 
might not be the case: 

P,;L=*i.?V.*=-ei (2) 

For the (p y;L or (p y;R ideally (p y;L = 2Acc and <Py :R = -2A<x. Allowing for small 
imperfections: 

<p y . L = 2Aa + e 2 ,<p x . R =-2Aa + £ 3 (3) 

It is assumed that the CCD is perfectly flat and has perfectly perpendicular image 
axes. The image formation is invariant for scaling of the triplet focal length, horizontal and 
vertical pixel size. Therefore, without loss of generality, the horizontal size of the pixels 
are chosen to be equal to 1 and the vertical size equal to R, the pixel aspect ratio. The 
positions of the projection frames PFivr (total CCD chip) relative to the lens frames UF^r 

are defined by a single vector Opf'Op"' Opf since refer to ^ same P h y sical 
camera. The first two numbers define the intersection of the lens optical axis with the total 
CCD (mis-positioning) and the third is the focal length f : 

Since a change of focal length in cameras is usually performed by movement of the 
lens rather than the CCD chip, h is modeled in (1) as a linear function off. 

h = a + bf (5) 

a and b are parameters estimated to establish a relationship between w and h. The 
orientation of the projection frames PFur (total CCD chip) relative to the lens frames 
LFur is defined by a single set of Euler angles (G x , 9 y , 6 Z ). 8 Z relates to the rotation of the 
projection frame. This is already modeled with <p z and thus 02= 0 is used. 

For the (p z : 



*n = ^ ( 6 ) 
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The 0x and 0 y model the non-orthogonal CCD placement with respect to the optical 
axis. Thus: 

0 x =6 8 ,0,=£ 9 ,0 z =O (7) 

Since mispositioning and misorientation of the CCD is incorporated in equations 
(4) and (7), lens distortion can be modeled more simply than in [6]. The radial distortion 
parameter K3 is used: 

K 3 =e l0 (8) 

The stereo camera model preferably contains the following parameters. For fixed 
calibration, there is a baseline B, a convergence angle Ace, a focal length/, a pixel aspect 
ratio R and ten error parameters 6i . . .. £10 which are assumed to be small. 

For self-calibration, the baseline B is discarded from the model by setting it to 1 
during the calibration [4]. Afterwards, it can be obtained by equation (1), provided that w, 
a and b have been determined before hand. 

ACQUISITION SCHEME 

Fig. 4 shows the complete scheme of acquisition. First, the hand-cam is mounted 
on the lens attachment. Then, a fixed calibration is performed for several values of 
convergence angle Aa and focal distance (zoom) f in order to obtain a, b and w. In 
addition, life-time constants such as the pixel aspect ratio R are obtained. The constants 
obtained will be invariant during the recording of the actual scene. 

Then the scene is recorded, during which any change in convergence angle and 
zoom arc allowed. Afterwards, the sequence is processed according to the left route in Fig. 
4. After correspondence estimation, self-calibration is applied [4]. The invariant (1) then 
enables 3-D model acquisition with the correct scale. 

Experiments 

A digital photocamera that takes 1024 x 768 images in JPEG format was used. Fig. 
5a shows images of the calibration plate (30) and Fig. 5b shows images of the actual scene 
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(31). Several fixed calibrations were performed, with different values of the convergence 
angle and focal distance. With the different values for fl,/and Act, a least squares 
technique was applied to estimate w, a and b. 

With the parameters from the Fixed calibration, a good 3-D model was obtained 
from the scene image (31) in Fig. 5b. After splitting the scene image in a left and right 
image pair, the images were rectified [5], as shown in Fig.6. All correspondences lie now 
on equal scan lines and 1-D disparity estimation was performed. A Markov Random Field 
disparity estimator [5] was used to obtain the disparity field shown in Fig. 7. After 
triangulation of all corresponding pixel pairs, the 3-D model (40) was obtained. Fig. 8 
shows the details in the facial area. 

The acquisition of 3-D scenes with a special device [7] enables stereo vision by a 
single hand-held camera (see Fig. 1). This system has several advantages. It is small and 
thus provides high user mobility, it needs only a single conventional recorder for storage 
and the use of a single camera is economical. 

Both camera calibration and correspondence estimation benefit from the device. 
Correspondence estimation relies on photometric similarity between corresponding pixels. 
With this device, there are no photometrical differences between left and right cameras. 
Further, left and right shutter synchronization is guaranteed by definition. For camera 
calibration, it was shown that a simpler stereo camera model could be used since the 
virtual left and right cameras share some physical properties from the single real camera. 
In addition, the device allows for self-calibration methods, while still providing a means 
for the capturing of absolute scale. 

The experiments showed that good 3-D models are obtained with the device. 
Currently the preferred method is a fixed calibration method, although a self-calibration 
method is also possible. The camera parameters obtained were well suited for the 
acquisition of a 3-D model. 

Accordingly, it is to be understood that the embodiments of the invention herein 
described are merely illustrative of the application of the principles of the invention. 
Reference herein to details of the illustrated embodiments is not intended to limit the 
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scope of the claims, which themselves recite those features regarded as essential to the 
invention. 
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What is claimed is: 



1 LA method of acquiring a three-dimensional scene using a single hand held camera 

2 having a lens, attached to a lens attachment comprising a right and a left inner 

3 mirror, both inner mirrors being located in front of the lens facing inwardly along 

4 an optical axis of the lens toward the lens, covering substantially all of a field of 

5 view of the lens, the inner mirrors being connected together at one edge at the 

6 optical axis of the lens, such that the two inner mirrors form an acute angle 

7 symmetrical about the optical axis of the lens, a right and a left outer mirror, both 

8 being larger than the inner mirrors and located outward of the inner mirrors, facing 

9 outward toward the subject, such that substantially all of the field of view of the 

10 inner mirrors is covered by the outer mirrors, the outer mirrors being pivotable 

1 1 about a vertical axis, a ganging mechanism for simultaneously pivoting the outer 

12 mirrors about their vertical axes in opposite senses, having a control input for 

13 actuating the ganging mechanism, such that a movement of the control input 

14 causes the right and left outer mirrors to pivot equally, oppositely and 

15 simultaneously, such that a movement of the control input in one direction causes 

16 the fields of view of the outer mirrors to converge, and a movement of the control 

17 input in the opposite direction causes the fields of view of the outer mirrors to 

18 diverge, and an adapter for moving the control input of the ganging mechanism in 

19 response to actuation of a focusing means of the lens, connected to the control 

20 input of the ganging mechanism and the focusing means of the lens, such that 

21 when the lens is focused at a selected distance, the field of view of the outer 

22 mirrors converges at the selected distance, comprising the steps of: 

23 a) deriving values for a, b, and w by performing a fixed calibration for at 

24 least one value of a convergence angle (Aa) and at least one focal 

25 distance (f) wherein w is defined as a distance between an optical 

26 center of the single hand held camera and an optical center of each 

27 of the virtual cameras, and a and b are parameters estimated to 

28 establish a relationship between w and h, wherein h is a distance 

29 between the optical center of the single hand held camera and an 

30 intersection of the two inner mirrors; 



WO 00/78055 PCT/US00/16044 



12 

31 b) recording an image of a scene using the single hand held camera and the 

32 lens attachment; 

33 c) processing the recorded image using the life-time constant, the value for 

34 the convergence angle (Aa) and the focal distance (0; and 

35 d) storing a processed three-dimensional image. 

1 2. The method of claim 1, wherein step (a) further comprises the steps of: 

2 i) recording at least one calibration plate; 

3 ii) splitting an image of the calibration plate into a left image and a right image; 

4 iii) detecting the calibration plate; and 

5 iv) calibrating the calibration plate. 

1 3. The method of claim 1, wherein step (c) further comprises the steps of: 

2 i) splitting the image of the scene into a left image and a right image; 

3 ii) performing correspondence estimation on the left image and the right image; 

4 iii) performing self-calibration; and 

5 iv) performing triangulation. 

1 4. The method of claim 1, further comprising the step of calculating a relationship 

2 between meters and angles using an equation B = 2w + 2 {h + w)sin 2Aa + e 0 

3 wherein B is defined as a camera baseline, and £o is defined as an error parameter. 

1 5. The method of claim 1 further comprising, prior to step (b), the step of defining five 

2 reference frames. 

1 6. The method of claim 5 wherein the five reference frames comprise: 

2 a) a stereo frame; 
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3 b) a left lens frame; 

4 c) a right lens frame; 

5 d) a left projection frame; and 

6 e) a right projection frame. 

1 7. The method of claim 6 wherein an orientation of the left and right lens frames are 

2 defined by two sets of Euler angles (q> x , (p y , <p z ). 

1 8. The method of claim 7, further comprising the step of introducing an equation that 

2 eliminates either cp x; L or (p x; R. 

1 9. The method of claim 8, wherein the equation is q> x; L + <Px ; r = 0. 

1 10. The method of claim 8, wherein when the Euler angles are not equal to zero, 

2 additional equations <p xL = £^<p XtR = -£ x are used, such that Z\ is a small error 

3 parameter. 

1 11. The method of claim 10, further comprising the step of calculating the equations 

2 <p y . L =2Aa + e 2 ,p, ;/f =-2Aa + e 3 . 

1 12. The method of claim 6 wherein a position of the left and right projection frames 

2 relative to the left and right lens frames are defined by a single vector 

3 Opf » Ow* OIf wherein Opf and Op"? defme m intersection of a lens optical 

4 axis with a total CCD and Q z £ is equal to the focal length f. 

1 13. The method of claim 1, further comprising the step of calculating a distance h 

2 between an optical center of the single hand held camera and an intersection of the 

3 inner mirrors using the equation h = a + bf . 

1 14. The method of claim 6 wherein an orientation of the left and right projection frames 

2 relative to the left and right lens frames is defined by a single set of Euler angles 

3 (6x1 0y» Gz)- 
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1 15. The method of claim 14, wherein 6 X and 9 y model the non-orthogonal CCD placement 

2 with respect to an optical axis such that 9 X =e 9 ,0 y =£ 97 6 z =0 where e 8 and £9 are 

3 defined as error parameters. 

1 16. The method of claim 14 wherein 8 Z relates to a rotation of the projection frame. 

1 17. The method of claim 1, further comprising the step of calculating a radial distortion 

2 parameter K3 using the equation 2sf 3 = e l0 , where e 10 is defined as an error 

3 parameter. 

1 18. The method of claim 1, where a plurality of parameters in step (a) further comprise a 

2 baseline B, a pixel aspect ratio R and ten error parameters, Ei, e 2 , e 3 , £4. £5, £6, £7. £8» 

3 £9, and £10, which are each assumed to be small. 

1 19. The method of claim 1, further comprising the step of self-calibrating the scene using 

2 a plurality of parameters including a baseline B equal to 1, a convergence angle 

3 Aoc, a focal length/, a pixel aspect ratio R and ten error parameters, Ei, E 2 , £3, £4, £5, 

4 £*, £7, £8, £9>,and E10, which are each assumed to be small. 



1 



20. A three-dimensional scene created by the method of claim 1. 



WO 00/78055 



PCT/US00/16044 




WO 00/78055 



2/6 



PCT/US00/16044 




WO 00/78055 



PCT/US00/16044 



Fig. 4 



3/6 



Recording of scene Recording of calibration plate 




Correspondence I 
estimation 1 



Self calibration 



^Fixed calibration Geometry 



Invariants 

Parameters ca | ibralion scene 

uw Geometry *J 

Parameters actual ^ analysis 




C— i 

Triangulation 



ft 

3-D scene model 



WO 00/78055 



PCT/US00/16044 




WO 00/78055 



PCT/USOO/16044 




Fig. 7 



WO 00/78055 



PCT/USOO/16044 



6/6 



Fig. 8 



40 



INTERNATIONAL SEARCH REPORT 


Int. UonaJ Application No 

PCT/US 00/16044 | 


A. CLASSIFICATION OF SUBJECT MATTER 

IPC 7 H04N13/02 




Accordxig to International Patent Classification (IPC) or to both national classification and IPC 




a FIELDS SEARCHED 


MHhiuni documentation ao ait hod (classification system followed by ciaaetflcatlon symbols) 

IPC 7 H04N 


Documentation aoarohed other than minimum documentation to the extent that such documents are Inct 


jded In the Holds ss arched 



ta base oonouf to d during the Intsmatlonal search (name of data base and, where 

EPO-Internal, WPI Data, PAJ, INSPEC 



a DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* Citation of document, with Indcation, where appropriate, of the relevant 



notevani to oasn no. 



EP 0 811 876 A (ZANEN PIETER 0) 

10 December 1997 (1997-12-10) 

page 5, column 8, line 36 -page 6, column 

10, line 3; figures 5,6 



1-20 



□ 



Further documents are 1st ad In the continuation of box C. 



Patortf brolly rrwmbsrai in Rated In aatmix. 



* SfDOCiftJ CeVtdQO{i94 Of cit 

'A* oocument defining the gsnsraf state of the art which Is not 

considered to be of particular relevance 
"E* eertlerdocumsnt but pubiahsd on or after the International 



V document which maytirow doubteonpriorlty dalm(s) or 
which la cftsd to estabssh the pubfloanon date of another 
elation or othsr specsal r easo n (as speoffled) 
■O" oocument leterring to an oral cHdoeure, use, exnfcftfonor 



•P* document puUshed prior to thajntemational filing date t 
later than the prior My date claimed 



T later document published after the international Wing date 
or priority date and not In conflict wth the appsoation but 
cted to understand the principle or theory underlying the 
Invention 

"X" document of particular rslavanoe; the claimed Invention 
cannot be conaidsred novel or cannot be considered to 
kivofve an Inventive step when the document Is taken alone 

"Y" oooument of particular relevance; the claimed Invention 

cannot be considered to Involve an Inventive step when the 
dpoumenjl la combined with one or more other such docu- 
ments, such combination being obvious to a person ski led 
in the art 

"A" docun tent member of the same patent femfly 



Dale of the actual com plot Ion of the international search 

4 September 2000 



n.1. —M ... m tMm 11 aJ Ik. I. I..,. .11 ..I i . , n h ■!■ mi mm 

lwhj or moling or ine iraemaiionei eearun repon 

11/09/2000 



i mating address of the ISA 

European Patent Office, P.B. 5618 F 
NL - 2280 HV* Rfjswljk 
Tel. (431-70) 340-2040. Tx. 31 661 epo nt, 
Fax: (+31-70) 340-0016 



Authorized officer 



De Paepe, W 



Foan PCT4SAJB10 (Moond atwet) (July 19*2) 



INTERNATIONAL SEARCH REPORT 

i on pdotiC f trolly mwiib#i • 



lnt> ion»l Application No 

PCT/US 00/16044 



Patent document 
cited in search report 



PuMcatton 



Patent family 
members) 



Publication 



EP 0811876 



10-12-1997 



US 
US 



5828913 A 
5883662 A 



27-10-1998 
16-03-1999 



Fowl PCT/IBAglO < p— n l fnty mux) (Ja» 1MB) 



